Repeat Consistency of a Result

We address the following issue: Suppose that we have taken a dataset consisting of a set of events. We do an analysis on this dataset, and obtain a result of interest. We then take the same dataset, and repeat the analysis, possibly with differences, to again obtain a result on the same matter of interest. How do we determine whether our two results are “consistent”? Let us first attempt to ensure that we understand the question. Consider the limit in which we repeat the identical analysis. In this case, the result should be identical. If it isn’t, there is an “inconsistency”. Clearly, “inconsistency” here means “mistake” – the assertion that the analyses are identical must be incorrect. Now consider the limit in which the first analysis includes a random selection of one-half of the events, and the second analysis includes a selection of precisely the half not used in the first analysis. The two analyses are otherwise identical. In this case the difference between the two results should be of purely statistical origin. “Consistency” in this case is somewhat less trivial to determine, because the results can differ due to statistical fluctuations. However, a conclusion of “inconsistency” is still an assertion that a “mistake” was made, up to whatever small probability is permitted for a large statistical fluctuation. The key point is that we are trying to evaluate whether any observed difference in the two analyses is an indication that there is something wrong with one or both analyses, or whether the difference is consistent with being due to statistical fluctuations.